You must have administrative privileges for a data source to add or delete servers in the Smart View Panel.
To delete a server from the Smart View Panel:
In the Shared Connections tree list, select the server to delete.
Right-click and select Remove server.