Details
-
Type:
Bug
-
Status: Resolved
-
Priority:
Major
-
Resolution: Won't Fix
-
Affects Version/s: 4.8.0
-
Fix Version/s: None
-
Component/s: app.filebrowser
-
Labels:None
-
Environment:
Hue : 4.8.0
HDFS : 3.x
Directory is EC Enabled.
Description
Problem Statement : Hue Upload doesnt work in EC Enabled Directory . Its fails with below Error In UI :
the runcupserver log says this :
[05/Nov/2020 22:17:49 -0800] exception ERROR Internal Server Error: /filebrowser/upload/file Traceback (most recent call last): File "/opt/cloudera/parcels/CDH-6.3.3-1.cdh6.3.3.p0.1796617/lib/hue/build/env/lib/python2.7/site-packages/Django-1.11.20-py2.7.egg/django/core/handlers/exception.py", line 41, in inner response = get_response(request) File "/opt/cloudera/parcels/CDH-6.3.3-1.cdh6.3.3.p0.1796617/lib/hue/build/env/lib/python2.7/site-packages/Django-1.11.20-py2.7.egg/django/core/handlers/base.py", line 249, in _legacy_get_response response = self._get_response(request) File "/opt/cloudera/parcels/CDH-6.3.3-1.cdh6.3.3.p0.1796617/lib/hue/build/env/lib/python2.7/site-packages/Django-1.11.20-py2.7.egg/django/core/handlers/base.py", line 178, in _get_response response = middleware_method(request, callback, callback_args, callback_kwargs) File "/opt/cloudera/parcels/CDH-6.3.3-1.cdh6.3.3.p0.1796617/lib/hue/build/env/lib/python2.7/site-packages/Django-1.11.20-py2.7.egg/django/middleware/csrf.py", line 300, in process_view request_csrf_token = request.POST.get('csrfmiddlewaretoken', '') File "/opt/cloudera/parcels/CDH-6.3.3-1.cdh6.3.3.p0.1796617/lib/hue/build/env/lib/python2.7/site-packages/Django-1.11.20-py2.7.egg/django/core/handlers/wsgi.py", line 126, in _get_post self._load_post_and_files() File "/opt/cloudera/parcels/CDH-6.3.3-1.cdh6.3.3.p0.1796617/lib/hue/build/env/lib/python2.7/site-packages/Django-1.11.20-py2.7.egg/django/http/request.py", line 302, in _load_post_and_files self._post, self._files = self.parse_file_upload(self.META, data) File "/opt/cloudera/parcels/CDH-6.3.3-1.cdh6.3.3.p0.1796617/lib/hue/build/env/lib/python2.7/site-packages/Django-1.11.20-py2.7.egg/django/http/request.py", line 261, in parse_file_upload return parser.parse() File "/opt/cloudera/parcels/CDH-6.3.3-1.cdh6.3.3.p0.1796617/lib/hue/build/env/lib/python2.7/site-packages/Django-1.11.20-py2.7.egg/django/http/multipartparser.py", line 261, in parse chunk = handler.receive_data_chunk(chunk, counters[i]) File "/opt/cloudera/parcels/CDH-6.3.3-1.cdh6.3.3.p0.1796617/lib/hue/desktop/libs/hadoop/src/hadoop/fs/upload.py", line 177, in receive_data_chunk self._file.write(raw_data) File "/opt/cloudera/parcels/CDH-6.3.3-1.cdh6.3.3.p0.1796617/lib/hue/desktop/libs/hadoop/src/hadoop/fs/upload.py", line 114, in write self._file.write(data) File "/opt/cloudera/parcels/CDH-6.3.3-1.cdh6.3.3.p0.1796617/lib/hue/desktop/libs/hadoop/src/hadoop/fs/webhdfs.py", line 981, in write self.append(data) File "/opt/cloudera/parcels/CDH-6.3.3-1.cdh6.3.3.p0.1796617/lib/hue/desktop/libs/hadoop/src/hadoop/fs/webhdfs.py", line 986, in append self._fs.append(self._path, data=data) File "/opt/cloudera/parcels/CDH-6.3.3-1.cdh6.3.3.p0.1796617/lib/hue/desktop/libs/hadoop/src/hadoop/fs/webhdfs.py", line 629, in append self._invoke_with_redirect('POST', path, params, data, headers) File "/opt/cloudera/parcels/CDH-6.3.3-1.cdh6.3.3.p0.1796617/lib/hue/desktop/libs/hadoop/src/hadoop/fs/webhdfs.py", line 858, in _invoke_with_redirect return resource.Resource(client).invoke(method, data=data, headers=headers) File "/opt/cloudera/parcels/CDH-6.3.3-1.cdh6.3.3.p0.1796617/lib/hue/desktop/core/src/desktop/lib/rest/resource.py", line 80, in invoke log_response=log_response) File "/opt/cloudera/parcels/CDH-6.3.3-1.cdh6.3.3.p0.1796617/lib/hue/desktop/core/src/desktop/lib/rest/resource.py", line 99, in _invoke clear_cookies=clear_cookies) File "/opt/cloudera/parcels/CDH-6.3.3-1.cdh6.3.3.p0.1796617/lib/hue/desktop/core/src/desktop/lib/rest/http_client.py", line 215, in execute raise self._exc_class(ex) WebHdfsException: 500 Server Error: Internal Server Error for url: http://myhue.example.com:14000/webhdfs/v1/user/admin/EC_test1/hive_check.diff.tmp?op=APPEND&doas=admin&data=true&user.name=hue {"RemoteException":{"message":"Append on EC file without new block is not supported. Use NEW_BLOCK create flag while appending file.","exception":"RemoteException","javaClassName":"org.apache.hadoop.ipc.RemoteException"}}
Reproduction Steps :
hdfs hdfs ec -enablePolicy -policy XOR-2-1-1024k hdfs ec -getPolicy -path /user/admin/EC_test1 hdfs ec -setPolicy -path /user/admin/EC_test1 -policy XOR-2-1-1024k
then login to Hue and try to upload an file in Abouve Created Directory. it fails with abouve error :
Root cause : http://hadoop.apache.org/docs/r3.1.0/hadoop-project-dist/hadoop-hdfs/HDFSErasureCoding.html#Limitations
Certain HDFS operations, i.e., hflush, hsync, concat, setReplication, truncate and append, are not supported on erasure coded files due to substantial technical challenges.
Internally Hue uses append functionality on Upload which fails with error : Append on EC file without new block is not supported. Use NEW_BLOCK create flag while appending file
Possible Fix/Proposed Fix : as appending to an file in EC enabled directory is not possible , i am proposing we create the file in Temp directory and then move it to the EC enabled Directory.
(already we are aware of temp directory via :
[hadoop]
[[hdfs_clusters]]
[[[default]]]
temp_dir=/tmp
) assuming the /tmp directory is not ec enabled and its configurable, we can uplaod file to there and later copy to the designated directory.
