Carlos Aguni

Highly motivated self-taught IT analyst. Always learning and ready to explore new skills. An eternal apprentice.


Hadoop HAS_DOWNSTREAM_IN_PIPELINE Troubleshoot

05 Jul 2022 »

https://blog.actorsfit.com/a?ID=01000-7c686c0c-cec7-4b3c-8535-fb5b70825a38

Data Node Exception when writing file To Hadoop

https://community.cloudera.com/t5/Support-Questions/Data-Node-Exception-when-writing-file-To-Hadoop/td-p/226287

From the above stacktraces it looks like, socket timeouts are set to very less values as 300ms. Hadoop’s default values are ReadTimeout=60000, WriteTimeout=8*60000

Please check below configurations in Datanode’s configs, “dfs.client.socket-timeout” “dfs.datanode.socket.write.timeout”

Anatomy of a File Write in HDFS

https://stackoverflow.com/questions/37663369/anatomy-of-a-file-write-in-hdfs

Understanding HDFS Recovery Processes (Part 1)