[Metrics] Add device index to the GPU metrics

[rkooo567]

Why are these changes needed?

Add device index and name to the GPU metrics.

The tags will be

{ip, GpuIndex, GpuDeviceName}

Related issue number

Checks

• I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[rkooo567]

Btw, any recommendation in e2e tests? In the unit test env, it is hard to test the real GPU usage because we don't have GPUs... should we add a single node release test? Or do we have the best practice testing GPU related code?

[rkooo567]

Also, I used {{DeviceName}}.{{index}}, {{ip}}, but it looks like the legend is not that pretty. Is there other recommendation?

Nvidia A10G.0, 127.0.0.1

[scottsun94]

RE legend, how about something like {{ip}}, gpu.{{index}}, {{DeviceName}} ?
It will look like this: 127.0.0.1, gpu.0, Nvidia A10G

There seems to be 4 scenarios:

1. single node, singe gpu
2. single node, multiple gpu
3. multiple nodes, single gpu per node
4. multiple nodes, multiple gpus per node

My considerations are:

• for 3, 4, users need to know ip first. Though people may want to see gpu index first in 2, but 3 and 4 are probably more important
• index itself may not be clear to users what it refers to. W&B and comet ml both use gpu.{{index}}. People should be familiar with it.
• Lastly, device name may not be that useful and may lead to a lot of UI repetition since people probably use the same GPU device across the cluster. However, we can test it out and remove it if that's the case.

[ericl]

Thanks for updating the rst files too

[ericl]

Suggested change

	["ip", "SessionName", "GpuDeviceName", "GpuIndex"],
	["ip", "SessionName", "GPUDeviceName", "GPUIndex"],

Shall we capitalize in this way consistently?

[rkooo567]

I am fine with that, but that will break the backward compatibility (since we documented these now... Not sure if we mentioned it is stable APIs).

I also found "ip" is not needed because we use Instance labels

[rkooo567]

I won't address it in this PR. Maybe I will make a follow up to use the same convention everywhere if we agree backward compatibility is not a concern.