Script started on 2021-04-12 00:13:49+0000 deeplearning@ML-RefVm-871628:~/mnist$ time python mnist-elastic.py.txt 2021-04-12 00:14:14.358362: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1 2021-04-12 00:14:15.613216: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcuda.so.1 2021-04-12 00:14:15.654278: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 0 with properties: pciBusID: 0001:00:00.0 name: Tesla K80 computeCapability: 3.7 coreClock: 0.8235GHz coreCount: 13 deviceMemorySize: 11.17GiB deviceMemoryBandwidth: 223.96GiB/s 2021-04-12 00:14:15.654327: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1 2021-04-12 00:14:15.656059: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.10 2021-04-12 00:14:15.657720: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcufft.so.10 2021-04-12 00:14:15.658003: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcurand.so.10 2021-04-12 00:14:15.659839: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusolver.so.10 2021-04-12 00:14:15.660866: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusparse.so.10 2021-04-12 00:14:15.664794: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.7 2021-04-12 00:14:15.666224: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1858] Adding visible gpu devices: 0 2021-04-12 00:14:15.666564: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN)to use the following CPU instructions in performance-critical operations: AVX2 FMA To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. 2021-04-12 00:14:15.673125: I tensorflow/core/platform/profile_utils/cpu_utils.cc:104] CPU Frequency: 2596995000 Hz 2021-04-12 00:14:15.673923: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x55da84ac71f0 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2021-04-12 00:14:15.673947: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2021-04-12 00:14:15.820291: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x55da84b53350 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2021-04-12 00:14:15.820331: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Tesla K80, Compute Capability 3.7 2021-04-12 00:14:15.821161: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 0 with properties: pciBusID: 0001:00:00.0 name: Tesla K80 computeCapability: 3.7 coreClock: 0.8235GHz coreCount: 13 deviceMemorySize: 11.17GiB deviceMemoryBandwidth: 223.96GiB/s 2021-04-12 00:14:15.821206: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1 2021-04-12 00:14:15.821239: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.10 2021-04-12 00:14:15.821278: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcufft.so.10 2021-04-12 00:14:15.821299: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcurand.so.10 2021-04-12 00:14:15.821317: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusolver.so.10 2021-04-12 00:14:15.821334: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusparse.so.10 2021-04-12 00:14:15.821356: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.7 2021-04-12 00:14:15.822660: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1858] Adding visible gpu devices: 0 2021-04-12 00:14:15.822707: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1 2021-04-12 00:14:16.192665: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1257] Device interconnect StreamExecutor with strength 1 edge matrix: 2021-04-12 00:14:16.192715: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1263] 0 2021-04-12 00:14:16.192730: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1276] 0: N 2021-04-12 00:14:16.194295: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1402] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10626 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0001:00:00.0, compute capability: 3.7) Model: "sequential" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= reshape (Reshape) (None, 784) 0 _________________________________________________________________ dense (Dense) (None, 512) 401920 _________________________________________________________________ dropout (Dropout) (None, 512) 0 _________________________________________________________________ dense_1 (Dense) (None, 512) 262656 _________________________________________________________________ dropout_1 (Dropout) (None, 512) 0 _________________________________________________________________ dense_2 (Dense) (None, 10) 5130 ================================================================= Total params: 669,706 Trainable params: 669,706 Non-trainable params: 0 _________________________________________________________________ Epoch 1/32 2021-04-12 00:14:17.334676: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.10 106/106 [==============================] - 46s 431ms/step - loss: 0.4623 - accuracy: 0.8541 - val_loss: 0.1644 - val_accuracy: 0.9497 Epoch 2/32 106/106 [==============================] - 46s 434ms/step - loss: 0.1955 - accuracy: 0.9403 - val_loss: 0.1228 - val_accuracy: 0.9625 Epoch 3/32 106/106 [==============================] - 46s 430ms/step - loss: 0.1356 - accuracy: 0.9572 - val_loss: 0.0885 - val_accuracy: 0.9738 Epoch 4/32 106/106 [==============================] - 46s 432ms/step - loss: 0.1074 - accuracy: 0.9666 - val_loss: 0.0710 - val_accuracy: 0.9798 Epoch 5/32 106/106 [==============================] - 46s 431ms/step - loss: 0.0901 - accuracy: 0.9721 - val_loss: 0.0676 - val_accuracy: 0.9788 Epoch 6/32 106/106 [==============================] - 46s 434ms/step - loss: 0.0812 - accuracy: 0.9751 - val_loss: 0.0624 - val_accuracy: 0.9818 Epoch 7/32 106/106 [==============================] - 46s 432ms/step - loss: 0.0713 - accuracy: 0.9768 - val_loss: 0.0573 - val_accuracy: 0.9828 Epoch 8/32 106/106 [==============================] - 46s 430ms/step - loss: 0.0633 - accuracy: 0.9799 - val_loss: 0.0521 - val_accuracy: 0.9840 Epoch 9/32 106/106 [==============================] - 45s 429ms/step - loss: 0.0606 - accuracy: 0.9810 - val_loss: 0.0572 - val_accuracy: 0.9840 Epoch 10/32 106/106 [==============================] - 45s 427ms/step - loss: 0.0573 - accuracy: 0.9821 - val_loss: 0.0501 - val_accuracy: 0.9853 Epoch 11/32 106/106 [==============================] - 45s 429ms/step - loss: 0.0486 - accuracy: 0.9846 - val_loss: 0.0491 - val_accuracy: 0.9853 Epoch 12/32 106/106 [==============================] - 46s 430ms/step - loss: 0.0476 - accuracy: 0.9847 - val_loss: 0.0508 - val_accuracy: 0.9852 Epoch 13/32 106/106 [==============================] - 46s 430ms/step - loss: 0.0465 - accuracy: 0.9844 - val_loss: 0.0467 - val_accuracy: 0.9860 Epoch 14/32 106/106 [==============================] - 45s 427ms/step - loss: 0.0429 - accuracy: 0.9865 - val_loss: 0.0455 - val_accuracy: 0.9877 Epoch 15/32 106/106 [==============================] - 46s 436ms/step - loss: 0.0411 - accuracy: 0.9873 - val_loss: 0.0447 - val_accuracy: 0.9873 Epoch 16/32 106/106 [==============================] - 45s 429ms/step - loss: 0.0389 - accuracy: 0.9875 - val_loss: 0.0465 - val_accuracy: 0.9865 Epoch 17/32 106/106 [==============================] - 45s 428ms/step - loss: 0.0376 - accuracy: 0.9878 - val_loss: 0.0486 - val_accuracy: 0.9863 Epoch 18/32 106/106 [==============================] - 45s 426ms/step - loss: 0.0319 - accuracy: 0.9896 - val_loss: 0.0449 - val_accuracy: 0.9873 Epoch 19/32 106/106 [==============================] - 45s 428ms/step - loss: 0.0324 - accuracy: 0.9889 - val_loss: 0.0449 - val_accuracy: 0.9893 Epoch 20/32 106/106 [==============================] - 45s 427ms/step - loss: 0.0333 - accuracy: 0.9891 - val_loss: 0.0408 - val_accuracy: 0.9897 Epoch 21/32 106/106 [==============================] - 45s 428ms/step - loss: 0.0316 - accuracy: 0.9896 - val_loss: 0.0424 - val_accuracy: 0.9880 Epoch 22/32 106/106 [==============================] - 45s 427ms/step - loss: 0.0309 - accuracy: 0.9903 - val_loss: 0.0446 - val_accuracy: 0.9893 Epoch 23/32 106/106 [==============================] - 45s 428ms/step - loss: 0.0286 - accuracy: 0.9909 - val_loss: 0.0444 - val_accuracy: 0.9890 Epoch 24/32 106/106 [==============================] - 46s 430ms/step - loss: 0.0287 - accuracy: 0.9905 - val_loss: 0.0406 - val_accuracy: 0.9897 Epoch 25/32 106/106 [==============================] - 45s 426ms/step - loss: 0.0273 - accuracy: 0.9913 - val_loss: 0.0423 - val_accuracy: 0.9893 Epoch 26/32 106/106 [==============================] - 45s 428ms/step - loss: 0.0280 - accuracy: 0.9908 - val_loss: 0.0406 - val_accuracy: 0.9890 Epoch 27/32 106/106 [==============================] - 45s 427ms/step - loss: 0.0257 - accuracy: 0.9918 - val_loss: 0.0423 - val_accuracy: 0.9893 Epoch 28/32 106/106 [==============================] - 45s 427ms/step - loss: 0.0240 - accuracy: 0.9922 - val_loss: 0.0414 - val_accuracy: 0.9890 Epoch 29/32 106/106 [==============================] - 45s 427ms/step - loss: 0.0231 - accuracy: 0.9923 - val_loss: 0.0439 - val_accuracy: 0.9898 Epoch 30/32 106/106 [==============================] - 45s 428ms/step - loss: 0.0234 - accuracy: 0.9926 - val_loss: 0.0434 - val_accuracy: 0.9888 Epoch 31/32 106/106 [==============================] - 45s 428ms/step - loss: 0.0239 - accuracy: 0.9922 - val_loss: 0.0430 - val_accuracy: 0.9888 Epoch 32/32 106/106 [==============================] - 46s 431ms/step - loss: 0.0230 - accuracy: 0.9925 - val_loss: 0.0416 - val_accuracy: 0.9892 real 24m47.424s user 24m56.129s sys 0m11.575s deeplearning@ML-RefVm-871628:~/mnist$ exit exit Script done on 2021-04-12 00:38:52+0000