Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add workflow to gen data #329

Open
wants to merge 244 commits into
base: feat/pull-v2-api
Choose a base branch
from

Conversation

ChenglimEar
Copy link
Contributor

@ChenglimEar ChenglimEar commented Nov 2, 2023

The purpose of this pull request is to add a GitHub workflow for the ETL that runs in Travis CI. This workflow will be used to help integrate the download logic for netfile v2 into make download.

The csvkit that was being used was a beta version (0.9.0) and it was making it difficult to upgrade the ETL logic. I updated csvkit to the latest 1.3.0 version in another branch (PR) to address upgrade issues and merged it to this branch. The change includes making use of SQL files for creating the tables. These SQL files originated from csvsql 0.9.0. These files make the schema independent of the version of csvsql used and can be modified going forward as needed to change the schema of the tables.

ckingbailey and others added 30 commits October 29, 2023 21:12
ChenglimEar and others added 10 commits April 13, 2024 16:32
This is an automated update by travis-ci at
Sat Apr 13 09:40:50 PDT 2024

[skip ci]
This is an automated update by travis-ci at
Sat Apr 13 09:47:48 PDT 2024

[skip ci]
This is an automated update by travis-ci at
Sat Apr 13 10:02:39 PDT 2024

[skip ci]
This is an automated update by travis-ci at
Sat Apr 13 17:13:22 PDT 2024

[skip ci]
ChenglimEar and others added 2 commits May 25, 2024 20:44
This is an automated update by travis-ci at
Tue May 28 21:29:40 PDT 2024

[skip ci]
@mikeubell
Copy link
Collaborator

Build diff from Commit 1e99f65:
diff --git a/build/_data/elections/oakland/2018-06-05.json b/build/_data/elections/oakland/2018-06-05.json
index b39b338..140316d 100644
--- a/build/_data/elections/oakland/2018-06-05.json
+++ b/build/_data/elections/oakland/2018-06-05.json
@@ -57,7 +57,7 @@
       "total_contributions": 15000.0
     },
     {
-      "name": "Service Employees International Union Local 1021 Issues PAC",
+      "name": "Oakland Athletics Baseball Company",
       "type": "Measure",
       "election_name": "oakland-june-2018",
       "total_contributions": 10000.0
diff --git a/build/_data/elections/oakland/2023-11-07.json b/build/_data/elections/oakland/2023-11-07.json
index 4e77c51..1e5045c 100644
--- a/build/_data/elections/oakland/2023-11-07.json
+++ b/build/_data/elections/oakland/2023-11-07.json
@@ -73,19 +73,19 @@
   ],
   "top_contributors_for_offices": [
     {
-      "name": "UA Local 342",
+      "name": "Service Employees International Union Local 1021 Candidate PAC",
       "type": "Office",
       "election_name": "oakland-2023",
       "total_contributions": 1200.0
     },
     {
-      "name": "Service Employees International Union Local 1021 Candidate PAC",
+      "name": "Oakland Education Association PAC",
       "type": "Office",
       "election_name": "oakland-2023",
       "total_contributions": 1200.0
     },
     {
-      "name": "Peralta Federation of Teachers COPE",
+      "name": "Families in Action for Justice",
       "type": "Office",
       "election_name": "oakland-2023",
       "total_contributions": 1200.0
diff --git a/build/_data/elections/oakland/2024-03-05.json b/build/_data/elections/oakland/2024-03-05.json
index db9ca88..a6261f2 100644
--- a/build/_data/elections/oakland/2024-03-05.json
+++ b/build/_data/elections/oakland/2024-03-05.json
@@ -48,13 +48,13 @@
   ],
   "top_contributors_for_offices": [
     {
-      "name": "Steven Von Stade",
+      "name": "Monique C Houston",
       "type": "Office",
       "election_name": "oakland-march-2024",
       "total_contributions": 600.0
     },
     {
-      "name": "Monique C Houston",
+      "name": "Steven Von Stade",
       "type": "Office",
       "election_name": "oakland-march-2024",
       "total_contributions": 600.0
diff --git a/build/_data/elections/oakland/2024-11-05.json b/build/_data/elections/oakland/2024-11-05.json
index 8440b5b..8bd5599 100644
--- a/build/_data/elections/oakland/2024-11-05.json
+++ b/build/_data/elections/oakland/2024-11-05.json
@@ -79,19 +79,19 @@
   ],
   "top_contributors_for_offices": [
     {
-      "name": "International Federation of Professional and Technical Engineers-Local 21 TJ Anthony PAC Fund",
+      "name": "International Association of Firefighters Local 55",
       "type": "Office",
       "election_name": "oakland-2024",
       "total_contributions": 2400.0
     },
     {
-      "name": "International Association of Firefighters Local 55",
+      "name": "International Federation of Professional and Technical Engineers-Local 21 TJ Anthony PAC Fund",
       "type": "Office",
       "election_name": "oakland-2024",
       "total_contributions": 2400.0
     },
     {
-      "name": "Weylin White",
+      "name": "Todd Scanlin",
       "type": "Office",
       "election_name": "oakland-2024",
       "total_contributions": 1800.0
diff --git a/build/_data/stats.json b/build/_data/stats.json
index 74f4147..35aa4a8 100644
--- a/build/_data/stats.json
+++ b/build/_data/stats.json
@@ -1,3 +1,3 @@
 {
-  "date_processed": "2024-04-13 00:08:40 -0700"
+  "date_processed": "2024-05-29 00:08:29 -0700"
 }
diff --git a/build/_data/totals.json b/build/_data/totals.json
index f6a1f4e..f82e1c3 100644
--- a/build/_data/totals.json
+++ b/build/_data/totals.json
@@ -1032,7 +1032,7 @@
         "total_contributions": 15000.0
       },
       {
-        "name": "Service Employees International Union Local 1021 Issues PAC",
+        "name": "Oakland Athletics Baseball Company",
         "type": "Measure",
         "election_name": "oakland-june-2018",
         "total_contributions": 10000.0
@@ -1385,19 +1385,19 @@
     ],
     "top_contributors_for_offices": [
       {
-        "name": "UA Local 342",
+        "name": "Service Employees International Union Local 1021 Candidate PAC",
         "type": "Office",
         "election_name": "oakland-2023",
         "total_contributions": 1200.0
       },
       {
-        "name": "Service Employees International Union Local 1021 Candidate PAC",
+        "name": "Oakland Education Association PAC",
         "type": "Office",
         "election_name": "oakland-2023",
         "total_contributions": 1200.0
       },
       {
-        "name": "Peralta Federation of Teachers COPE",
+        "name": "Families in Action for Justice",
         "type": "Office",
         "election_name": "oakland-2023",
         "total_contributions": 1200.0
@@ -1500,13 +1500,13 @@
     ],
     "top_contributors_for_offices": [
       {
-        "name": "Steven Von Stade",
+        "name": "Monique C Houston",
         "type": "Office",
         "election_name": "oakland-march-2024",
         "total_contributions": 600.0
       },
       {
-        "name": "Monique C Houston",
+        "name": "Steven Von Stade",
         "type": "Office",
         "election_name": "oakland-march-2024",
         "total_contributions": 600.0
@@ -1646,19 +1646,19 @@
     ],
     "top_contributors_for_offices": [
       {
-        "name": "International Federation of Professional and Technical Engineers-Local 21 TJ Anthony PAC Fund",
+        "name": "International Association of Firefighters Local 55",
         "type": "Office",
         "election_name": "oakland-2024",
         "total_contributions": 2400.0
       },
       {
-        "name": "International Association of Firefighters Local 55",
+        "name": "International Federation of Professional and Technical Engineers-Local 21 TJ Anthony PAC Fund",
         "type": "Office",
         "election_name": "oakland-2024",
         "total_contributions": 2400.0
       },
       {
-        "name": "Weylin White",
+        "name": "Todd Scanlin",
         "type": "Office",
         "election_name": "oakland-2024",
         "total_contributions": 1800.0

@ChenglimEar ChenglimEar changed the title WIP: Add workflow to gen data Add workflow to gen data May 30, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants