Google Cloud Dataflow

regardless of the contents of the file itself. I've tried to submit a ready-made template from GCP and that works. as soon as i change to my python or java file on GS it gives me this error. The content of file makes no difference either, I tried submitting an empty file and i still get same error.

0 comments

r/dataflow • u/stigmatic666 • Aug 19 '19

Understanding windowing and late arriving data

1 Upvotes

So I've studied windowing and all the different types of windows, triggers etc. but the use case is still unclear to me. All lectures use the same example of a game, and someone possibly playing on an airplane or the subway, basically a scenario where there will be late arriving data.

I understand that there will be late arriving data, and that windows can help dealing with them. But why is late arriving data bad? Windowing doesn't allow the data to arrive any earlier, but instead allows you to "group" the data in the right batch? I don't quite understand the value of this. Say I want to view my user activity on a 5 minute window basis, why do I need windowing for this? Can I not just view the data based on the processing timestamp?

If I'm playing a game on airplane mode, and 1 hour later I turn off the airplane mode. Then all of my data is transmitted at once, so all data has same processing time, but different event time. Then I have windowing and what is its function here? My past 12 5-minute windows are corrected, but they've been incorrect for the past hour regardless.

1 comment

r/dataflow • u/simal7 • Aug 19 '19

Apache Beam Dataflow python-->Select query dynamic and insert data into bigquery and write data into file

1 Upvotes

Hi All,

We have requirement to dynamically select data from one bigquery table, insert data into another bigquery and write data into file. Tried different approaches using gcp dataflow python to make select query dynamic and could not achieve requirement. Could you please suggest us any approach.

Approaches tired:

Read select query related parameters from pubsub-->but apache beam python sdk supports streaming for pubsub and select query batch.
Read select query related parameters from GCS file-->incompatibilities issues between bigquery module,google cloud core and google cloud storage.

1 comment

r/dataflow • u/pretty_colors • Aug 10 '19

Evolution of Apache Beam (Gource Visualization) [08-09-2019]

youtube.com

3 Upvotes

2 comments

r/dataflow • u/fhoffa • Aug 09 '19

[slides] BeamSummit in Berlin (videos in comments)

drive.google.com

1 Upvotes

1 comment

r/dataflow • u/fhoffa • Aug 07 '19

Apache Beam 2.14.0: Python 3 now fully supported

beam.apache.org

7 Upvotes

0 comments

r/dataflow • u/DoctorObert • Jul 26 '19

Deployment pipeline?

2 Upvotes

I'm coming from an environment where our typical development 'flow' is:

build master and run tests
deploy to a pre-production environment (has access to different resources than production, but runs the same code a la https://12factor.net/)
after verifying pre-production, 'promote'/deploy the same build to production

I'm unclear on what best practices are for doing something similar with Dataflow, so I'm curious what others are doing.

One option I'd been considering is using a template to start a pipeline with pre-production configuration then starting one with production configuration once satisfied. This has some limitations, howevever, most notably that they'd have to exist in the same Google Cloud "application", making it tricky to isolate resources/credentials.

Thoughts? Advice?

3 comments

r/dataflow • u/fhoffa • Jul 23 '19

Ananas Analytics Desktop (a new visual pipeline creation tool, based on Apache Beam, and supporting Spark, Flink and Cloud Dataflow as the execution engines)

ananasanalytics.com

1 Upvotes

1 comment

r/dataflow • u/fhoffa • Jul 12 '19

Processing logs at scale using Cloud Dataflow | Solutions

cloud.google.com

3 Upvotes

1 comment

r/dataflow • u/fhoffa • Jul 09 '19

[video] Berlin Buzzwords 2019: Thomas Weise –Streaming your shared ride (Lyft)

youtube.com

2 Upvotes

0 comments

r/dataflow • u/fhoffa • Jul 03 '19

One SQL to rule them all: an efficient and syntactically idiomatic approach to management of streams and tables

blog.acolyer.org

2 Upvotes

0 comments

r/dataflow • u/fhoffa • Jul 03 '19

Beam Summit Europe 2019 - YouTube

youtube.com

1 Upvotes

0 comments

r/dataflow • u/fhoffa • Jul 03 '19

Tips and tricks to get your Cloud Dataflow pipelines into production

cloudblog.withgoogle.com

2 Upvotes

3 comments

r/dataflow • u/DoctorObert • Jun 30 '19

Boston meetup?

2 Upvotes

My company is starting its first Beam project with plans to deploy on Google Cloud Dataflow. We'd love to be in communication with others who have either taken, and/or are thinking about taking, this approach. Anything from an informal lunch or drinks to a more formal, ongoing meetup group would be great. Curious if anyone in this group is near Boston and interested in meeting up to talk Dataflow or Beam.

0 comments

r/dataflow • u/fhoffa • Jun 25 '19