Log lines have inconsistent log levels and formats #602
Description
error logs are logged with info level, making it difficult to search through log lines to see what failed vs what are just informational lines
{"level":"info","ts":"2024-02-14T21:49:12.387Z","logger":"controller.route","caller":"controllers/route_controller.go:155","msg":"reconcile er ror","name":"traffic-exemplar","message":"error during service synthesis failed ServiceManager.Upsert REDACTED due to ValidationException: customDomainName is required when certificateArn is provided.\n{\n RespMetadata: {\n StatusCode: 400,\n RequestID : \"1e3f12ab-4038-4212-9d94-f2f186be5c1b\"\n },\n FieldList: [{\n Message: \"customDomainName is required when certificateArn is provid ed.\",\n Name: \"certificateArn\"\n }],\n Message_: \"customDomainName is required when certificateArn is provided.\",\n Reason: \"f ieldValidationFailed\"\n}"} using version 1.0.1 but this hasn't changed in main branch either: https://github.com/aws/aws-application-networking-k8s/blob/main/pkg/controllers/route_controller.go#L155
Can warn/error logs be used more consistently? eg warn for errors that are ignored and error for errors that actually cause a reconciliation failure?
Another idea is to use some consistent messaging in log lines to indicate that something errored. eg if "failed_reconcile" will always include the error message of why the reconciliation failed that would be easy to search.