Advertisement
Not a member of Pastebin yet?
Sign Up,
it unlocks many cool features!
- messages = spark.read.parquet("/user/madaxell/data/events/events.parquet")\
- .where("event_type = 'reaction'")\
- .select(F.col("event.message_id").alias("message_id"),
- F.col("event.reaction_from").alias("user_id"),
- #F.explode(F.col("event.tags")).alias("tag"),
- F.col("event.tags").alias("tags"),
- F.col("event.reaction_type").alias("reaction"))\
- .where("tags IS NOT NULL")
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement