Questions tagged [apache-airflow]
Airflow is a platform created by the community to programmatically author, schedule and monitor workflows.
12 questions
0 votes
0 answers
127 views
Registering Custom Plugins and DAG Serialization in Airflow
I'm having trouble getting a custom plugin to work properly with the Web UI in Apache Airflow. The DAG appears in the list of DAGs but attempting to view it via the console returns a 500 server error. ...
0 votes
0 answers
108 views
Clould composer airflow dag dataflow job error
I am running dag for dataflow java operator in composer airflow, default_args = { 'retries': 0, 'start_date': airflow.utils.dates.days_ago(0), 'owner': 'Airflow_TEST', '...
google-cloud-platform
0 votes
1 answer
766 views
Google Cloud Composer 2 Environment won't become healthy
I am trying to create a Google Cloud Composer 2 environment in my project, but it fails to become healthy. I am creating it with the default settings and a service account that has the Cloud Composer ...
0 votes
1 answer
690 views
How to check current host os in Docker compose?
For linux and airflow containers, we have to provide host user id to the container: "On Linux, the quick-start needs to know your host user id and needs to have group id set to 0. Otherwise the ...
- 113
1 vote
0 answers
132 views
How can I get Airflow on Linux run a VBS script that is automatically downloading SAP data?
I have an Airflow installed on CentOS and I am able to run scripts/commando's on my Windows 10 machine. However, I have a VBS script that starts Excel and SAP Logon and then automatically logins, ...
- 11
0 votes
0 answers
34 views
Which server specs to look for for a Scheduler in a data stack?
I'm in the process of building a data stack for a small company : the choice has been made to have a UNIX server running as a "Scheduler". The goal of this scheduler is just to extract data ...
- 1
0 votes
1 answer
864 views
How should "CPU usage per node" be interpreted in Google Cloud monitoring?
In the monitoring tab for Composer (Airflow) on Google Cloud there is a graph showing "CPU usage per node". How should the values in this graph be interpreted? What value would indicate that ...
0 votes
1 answer
1k views
Running a task in parallel in multiple machines in GCP and orchestrating it
I have a python program for scraping that needs a lot of time to run. To parallelize it, I have modified the code so that the program can run in parallel in different machines. I also created a docker ...
-1 votes
1 answer
2k views
How to set smtp_password in GCP Cloud Composer for configuring third-party SMTP services
I can use only G Suit SMTP mail server due to company security policy. I want to configuring 3rd-party SMTP service for using G Suit mail service but I can't set "smtp smtp_password" config ...
0 votes
1 answer
1k views
Google Cloud Composer "Error: Server Error"
My Google Cloud Composer is not working since yesterday. I can't access the airflow interface, returning: Error: Server Error The server encountered a temporary error and could not complete your ...
0 votes
1 answer
718 views
ETL to bigquery using airflow without have permission cloud storage/ cloud sql
i have done ETL from MySql to bigQuery with python, but because i haven't permission to connect google cloud storage/ cloud sql, i must dump data and partition that by last date, this way easy but ...
2 votes
1 answer
18k views
Airflow: Can't connect to ('0.0.0.0', 8080)
I am trying to do some stuff with Airflow.I have installed it with pip(latter in a virtual) but problem is when I want to start webserver. airflow webserver -p 8080 [2017-07-29 12:20:45,913] [4585] {...
- 123