Skip to Content
0 Votes

Changes for page Friday 14 February 2025

Last modified by dennis yoshikawa on 2025/02/14 01:23
From version 4.19
edited by Haekal Yusril Faizin
on 2025/02/13 12:25
Change comment: There is no comment for this version
To version 6.5
edited by dennis yoshikawa
on 2025/02/14 01:18
Change comment: There is no comment for this version

Summary

Details

Page properties
Author
... ... @@ -1,1 +1,1 @@
1 -XWiki.haekalfaizin
1 +XWiki.dennis
Content
... ... @@ -10,25 +10,47 @@
10 10  What Have You Done?
11 11  {{/success}}
12 12  
13 -1. xxx
14 -1. xxx
15 -1. xxx
13 +* Airflow-airbyte maintenance
16 16  
15 +~1. create task for delete gcs folder table dharmadexa
16 +2. ⁠change incremental method in airbyte for tb)
17 +
18 +* Dkonsul
19 +
20 +~1. Deploy transformation code to production.
21 +2. Final QC with partnership team.
22 +
23 +* ⁠*D2D*
24 +
25 +~1. enhance logic stickness rate (mart)
26 +2. ⁠weekly regrup w/product
27 +
28 +* MCN
29 +
30 +~1. troubleshoot on DAG and resource
31 +2. ⁠repointing new database for all pipeline existing
32 +
33 +* Screening
34 +
35 +~1. regroup w/PM and DA for validate data and discuss funneling data
36 +2. enhance logic fact table dharmadexa
37 +3. ⁠preparation dharmadexa data phase 3 (funneling)
38 +4. ⁠testing geo.py to get city from long lat
39 +5. ⁠migrasi dataset cleaned_screening_merck.dim_screening_stunting to CH
40 +
17 17  {{warning}}
18 18  What Issues You Have?
19 19  {{/warning}}
20 20  
21 -1. xxx
22 -1. xxx
23 -1. xxx
45 +1. That's difficulty in finding open source data quality tools is that most of them are paid.
24 24  
25 25  {{info}}
26 26  What Next You Will Do? (Optional)
27 27  {{/info}}
28 28  
51 +1. Exp
29 29  1. xxx
30 30  1. xxx
31 -1. xxx
32 32  
33 33  **What Support You Need? (Optional)**
34 34  
... ... @@ -91,18 +91,18 @@
91 91  
92 92  **AUTOMARK**
93 93  
94 -1. Deploy, evaluate and making documentation of **Screening Dharma Dexa API** to be accessed by Tech team. The API is deployed at [[https:~~/~~/datalake.ptgue.com/v1/users_dd>>https://datalake.ptgue.com/v1/users_dd]] and the **documentation is already sent to Tech team**. The API could be accessed using prompt to retrieve user that relevant to the promptPer 13 Feb 2024, **the RAG Accuracy is 87,50%**
116 +1. Deploy, evaluate and making documentation of **Screening Dharma Dexa API** to be accessed by Tech team. The API is deployed at [[https:~~/~~/datalake.ptgue.com/v1/users_dd>>https://datalake.ptgue.com/v1/users_dd]] and the **documentation is already sent to Tech team**. **The API could be accessed using prompt** to retrieve user that relevant to the prompt. Per 13 Feb 2024, **the RAG Accuracy is 87,50%**
95 95  1. Deploy, evaluate and making documentation of **master user GUE Ecosystem** to be accessed by Tech team. The API is deployed at [[https:~~/~~/datalake.ptgue.com/v1/users>>https://datalake.ptgue.com/v1/users_dd]] and the **documentation is already sent to Tech team**. **The API is already tested by Tech team and have no issues.**
96 96  
97 97  **MCN**
98 98  
99 -1. Fixing resource spike issue caused by scraping schedule. The issue is already fixed and pass the test for 3 days (wednesday, thursday, and friday). Scraping time is decreased from 1 minute per user to the maximum of +-20 seconds per user. (Pairing with Syifa-DE)
100 -1. Repoint, re-align, and redesign the pipeline and database that being consumed for scrapping. (Pairing with Syifa-DE)
121 +1. Fixing resource spike issue caused by scraping schedule. The issue is already fixed and pass the test for 3 days (wednesday, thursday, and friday). **Scraping time is decreased from 1 minute per user to the maximum of +-20 seconds per user.** **(Pairing with Syifa-DE)**
122 +1. **Repoint, re-align, and redesign the pipeline and database** that being consumed for scrapping.** (Pairing with Syifa-DE)**
101 101  
102 102  **Automation**
103 103  
104 -1. Implement compliance mapping with the newest data from compliance team
105 -1. Data, flow, and script validation for user report performance doctor. The report sample is already validated by dr.Astrid.
126 +1. **Implement compliance mapping** with the newest data from compliance team
127 +1. Data, flow, and script validation for user report performance doctor. **The sample report is already validated by dr.Astrid**
106 106  
107 107  **Others**
108 108  
... ... @@ -114,19 +114,53 @@
114 114  What Issues You Have?
115 115  {{/warning}}
116 116  
117 -1. xxx
118 -1. xxx
119 -1. xxx
139 +1. There is no info regarding database repointing of MCN from tech team. So, the data is not updated. Already solved by coordinating with Product Team
140 +1. Need to enhance the RAG accuracy to around 90%
120 120  
121 121  {{info}}
122 122  What Next You Will Do? (Optional)
123 123  {{/info}}
124 124  
125 -1. xxx
126 -1. xxx
127 -1. xxx
146 +1. Increase RAG accuracy by adding more train query LLM
147 +1. Start daily recurring scraping for tiktok dashboard
148 +1. Start dkonsul insight next week
128 128  
129 129  **What Support You Need? (Optional)**
130 130  
131 -
152 +
153 +=== Summary ===
154 +
155 +Berikut ringkasan laporan mingguan dari 14 Februari 2025 yang ditulis oleh Haekal Yusril Faizin. Laporan ini merangkum aktivitas dan isu dari tim Data Engineer, Data Analyst, dan Data Analyst & AI.
156 +
157 +**Data Analyst**
158 +
159 +* Menyelesaikan fase kedua Dharma Dexa.
160 +* Memindahkan visualisasi data dan analitik dari Looker ke Metabase.
161 +* Mendesain ulang data DKonsul untuk struktur dan organisasi yang lebih baik.
162 +* Menangani berbagai permintaan data on-demand.
163 +
164 +**Data Analyst & AI**
165 +
166 +* Menerapkan, mengevaluasi, dan mendokumentasikan API Screening Dharma Dexa untuk diakses oleh tim Tech.
167 +* Menerapkan, mengevaluasi, dan mendokumentasikan master user GUE Ecosystem untuk diakses oleh tim Tech.
168 +* Memperbaiki masalah lonjakan sumber daya yang disebabkan oleh jadwal scraping.
169 +* Merepoint, menyelaraskan ulang, dan mendesain ulang pipeline dan database yang digunakan untuk scraping.
170 +* Menerapkan pemetaan kepatuhan dengan data terbaru dari tim kepatuhan.
171 +* Memvalidasi data, alur, dan skrip untuk kinerja laporan pengguna dokter.
172 +
173 +**Isu**
174 +
175 +* Pilihan bagan dan fleksibilitas yang terbatas dalam kustomisasi Metabase.
176 +* Kurangnya informasi mengenai database repointing MCN dari tim teknologi.
177 +* Perlu meningkatkan akurasi RAG menjadi sekitar 90%.
178 +
179 +**Langkah Selanjutnya**
180 +
181 +* Melanjutkan desain ulang data DKonsul dan migrasi ke Metabase.
182 +* Melanjutkan dengan Dharma Dexa Fase 3 dan mengembangkan Dasbor Pelacak Kunjungan AppSheet MCN.
183 +* Meningkatkan akurasi RAG, memulai scraping berulang harian untuk dasbor TikTok, dan memulai wawasan DKonsul minggu depan.
184 +
185 +**Dukungan yang Dibutuhkan**
186 +
187 +* Validasi data dan peningkatan penyaringan Dharma Dexa.
132 132  )))