• k8s-merge-robot's avatar
    Merge pull request #24808 from screeley44/gluster_errors · 0b7f8e5b
    k8s-merge-robot authored
    Automatic merge from submit-queue
    
    read gluster log to surface glusterfs plugin errors properly in describe events
    
    glusterfs.go does not properly expose errors as all mount errors go to a log file, I propose we read the log file to expose the errors without asking the users to 'go look at this log'
    
    This PR does the following:
    1.  adds a gluster option for log-level=ERROR to remove all noise from log file
    2.  change log file name and path based on PV + Pod name - so specific per PV and Pod
    3.  create a utility to read the last two lines of the log file when failure occurs
    
    old behavior:
    ```
      13s	13s	1	{kubelet 127.0.0.1}		Warning	FailedMount	Unable to mount volumes for pod "bb-gluster-pod2_default(34b18c6b-070d-11e6-8e95-52540092b5fb)": glusterfs: mount failed: Mount failed: exit status 1
    Mounting arguments: 192.168.234.147:myVol2 /var/lib/kubelet/pods/34b18c6b-070d-11e6-8e95-52540092b5fb/volumes/kubernetes.io~glusterfs/pv-gluster glusterfs [log-file=/var/lib/kubelet/plugins/kubernetes.io/glusterfs/pv-gluster/glusterfs.log]
    Output: Mount failed. Please check the log file for more details.
    ```
    
    improved behavior: (updated after suggestions from community)
    ```
      34m		34m		1	{kubelet 127.0.0.1}			Warning		FailedMount	Unable to mount volumes for pod "bb-multi-pod1_default(e7d7f790-0d4b-11e6-a275-52540092b5fb)": glusterfs: mount failed: Mount failed: exit status 1
    Mounting arguments: 192.168.123.222:myVol2 /var/lib/kubelet/pods/e7d7f790-0d4b-11e6-a275-52540092b5fb/volumes/kubernetes.io~glusterfs/pv-gluster2 glusterfs [log-level=ERROR log-file=/var/lib/kubelet/plugins/kubernetes.io/glusterfs/pv-gluster2/bb-multi-pod1-glusterfs.log]
    Output: Mount failed. Please check the log file for more details.
    
     the following error information was pulled from the log to help resolve this issue: 
    [2016-04-28 14:21:29.109697] E [socket.c:2332:socket_connect_finish] 0-glusterfs: connection to 192.168.123.222:24007 failed (Connection timed out)
    [2016-04-28 14:21:29.109767] E [glusterfsd-mgmt.c:1819:mgmt_rpc_notify] 0-glusterfsd-mgmt: failed to connect with remote-host: 192.168.123.222 (Transport endpoint is not connected)
    
    ```
    
    also this PR is alternate approach to :  #24624 
    
    0b7f8e5b
Name
Last commit
Last update
..
admission Loading commit data...
api Loading commit data...
apimachinery Loading commit data...
apis Loading commit data...
apiserver Loading commit data...
auth Loading commit data...
capabilities Loading commit data...
client Loading commit data...
cloudprovider Loading commit data...
controller Loading commit data...
conversion Loading commit data...
credentialprovider Loading commit data...
dns Loading commit data...
fieldpath Loading commit data...
fields Loading commit data...
genericapiserver Loading commit data...
healthz Loading commit data...
httplog Loading commit data...
hyperkube Loading commit data...
kubectl Loading commit data...
kubelet Loading commit data...
kubemark Loading commit data...
labels Loading commit data...
master Loading commit data...
metrics Loading commit data...
probe Loading commit data...
proxy Loading commit data...
quota Loading commit data...
registry Loading commit data...
runtime Loading commit data...
security Loading commit data...
securitycontext Loading commit data...
serviceaccount Loading commit data...
ssh Loading commit data...
storage Loading commit data...
types Loading commit data...
ui Loading commit data...
util Loading commit data...
version Loading commit data...
volume Loading commit data...
watch Loading commit data...
OWNERS Loading commit data...