Troubleshooting - Deploying installer-provisioned clusters on bare metal | Installing

About
- Welcome
- Kubernetes overview
- Legal notice
What's new?
- New features and enhancements
- Deprecated features
Release notes
- OpenShift Container Platform 4.6 release notes
Architecture
- Architecture overview
- Product architecture
- Installation and update
- The control plane
- Understanding OKD development
- Fedora CoreOS
- Using ArgoCD
Installing
- Mirroring images for a disconnected installation
- Installing on AWS
- Installing on Azure
- Installing on GCP
- Installing on bare metal
- Deploying installer-provisioned clusters on bare metal
- Installing on OpenStack
- Installing on RHV
- Installing on vSphere
- Installing on VMC
- Installing on any platform
  - Installing a cluster on any platform
- Installation configuration
- Validating an installation
- Troubleshooting installation issues
Updating clusters
- Updating clusters overview
- Understanding the OpenShift Update Service
- Installing and configuring the OpenShift Update Service
- Understanding upgrade channels
- Updating a cluster using the web console
- Updating a cluster using the CLI
Post-installation configuration
- Post-installation configuration overview
- Machine configuration tasks
- Cluster tasks
- Node tasks
- Network configuration
- Storage configuration
- Preparing for users
- Configuring alert notifications
Web console
- Web console overview
- Accessing the web console
- Viewing cluster information
- Configuring the web console
- Customizing the web console
- Disabling the web console
CLI tools
- CLI tools overview
- OpenShift CLI (oc)
- Developer CLI (odo)
- Helm CLI
  - Getting started with Helm
  - Configuring custom Helm chart repositories
- Knative CLI (kn) for use with OpenShift Serverless
- opm CLI
Security and compliance
- Security and compliance overview
- Container security
- Configuring certificates
- Certificate types and descriptions
- Compliance Operator
- File Integrity Operator
- Viewing audit logs
- Configuring the audit log policy
- Configuring TLS security profiles
- Configuring seccomp profiles
- Allowing JavaScript-based access to the API server from additional hosts
- Encrypting etcd data
- Scanning pods for vulnerabilities
Authentication and authorization
- Authentication and authorization overview
- Understanding authentication
- Configuring the internal OAuth server
- Configuring OAuth clients
- Understanding identity provider configuration
- Configuring identity providers
- Using RBAC to define and apply permissions
- Removing the kubeadmin user
- Understanding and creating service accounts
- Using service accounts in applications
- Using a service account as an OAuth client
- Scoping tokens
- Using bound service account tokens
- Managing security context constraints
- Impersonating the system:admin user
- Syncing LDAP groups
Networking
- Understanding networking
- Accessing hosts
- Networking Operators overview
- Understanding the Cluster Network Operator
- Understanding the DNS Operator
- Understanding the Ingress Operator
- Configuring the node port service range
- Using SCTP
- Configuring PTP hardware
- Network policy
  - About network policy
  - Creating a network policy
  - Viewing a network policy
  - Editing a network policy
  - Deleting a network policy
  - Defining a default network policy for projects
  - Configuring multitenant network policy
- Multiple networks
  - Understanding multiple networks
  - Configuring an additional network
  - Attaching a pod to an additional network
  - Removing a pod from an additional network
  - Editing an additional network
  - Removing an additional network
- Hardware networks
  - About Single Root I/O Virtualization (SR-IOV) hardware networks
  - Installing the SR-IOV Operator
  - Configuring the SR-IOV Operator
  - Configuring an SR-IOV network device
  - Configuring an SR-IOV Ethernet network attachment
  - Configuring an SR-IOV InfiniBand network attachment
  - Adding pod to an SR-IOV network
  - Using high performance multicast
  - Using DPDK and RDMA
  - Uninstalling the SR-IOV Operator
- OpenShift SDN default CNI network provider
  - About the OpenShift SDN default CNI network provider
  - Configuring egress IPs for a project
  - Configuring an egress firewall for a project
  - Viewing an egress firewall for a project
  - Editing an egress firewall for a project
  - Removing an egress firewall from a project
  - Considerations for the use of an egress router pod
  - Deploying an egress router pod in redirect mode
  - Deploying an egress router pod in HTTP proxy mode
  - Deploying an egress router pod in DNS proxy mode
  - Configuring an egress router pod destination list from a config map
  - Enabling multicast for a project
  - Disabling multicast for a project
  - Configuring multitenant isolation
  - Configuring kube-proxy
- OVN-Kubernetes default CNI network provider
  - About the OVN-Kubernetes network provider
  - Migrating from the OpenShift SDN cluster network provider
  - Rolling back to the OpenShift SDN cluster network provider
  - Configuring an egress firewall for a project
  - Viewing an egress firewall for a project
  - Editing an egress firewall for a project
  - Removing an egress firewall from a project
  - Configuring an egress IP address
  - Assigning an egress IP address
  - Enabling multicast for a project
  - Disabling multicast for a project
  - Configuring hybrid networking
- Configuring Routes
  - Route configuration
  - Secured routes
- Configuring ingress cluster traffic
  - Overview
  - Configuring ExternalIPs for services
  - Configuring ingress cluster traffic using an Ingress Controller
  - Configuring ingress cluster traffic using a load balancer
  - Configuring ingress cluster traffic on AWS using a Network Load Balancer
  - Configuring ingress cluster traffic using a service external IP
  - Configuring ingress cluster traffic using a NodePort
- Configuring the cluster-wide proxy
- Configuring a custom PKI
- Load balancing on OpenStack
- Associating secondary interfaces metrics to network attachments
Storage
- Storage overview
- Understanding ephemeral storage
- Understanding persistent storage
- Configuring persistent storage
- Using Container Storage Interface (CSI)
- Expanding persistent volumes
- Dynamic provisioning
Registry
- Registry overview
- Image Registry Operator in OKD
- Setting up and configuring the registry
- Accessing the registry
- Exposing the registry
Operators
- Operators overview
- Understanding Operators
- User tasks
  - Creating applications from installed Operators
  - Installing Operators in your namespace
- Administrator tasks
- Developing Operators
- Red Hat Operators reference
Builds
- Understanding image builds
- Understanding build configurations
- Creating build inputs
- Managing build output
- Using build strategies
- Custom image builds with Buildah
- Performing basic builds
- Triggering and modifying builds
- Performing advanced builds
- Using Red Hat subscriptions in builds
- Securing builds by strategy
- Build configuration resources
- Troubleshooting builds
- Setting up additional trusted certificate authorities for builds
Images
- Overview of images
- Configuring the Cluster Samples Operator
- Using the Cluster Samples Operator with an alternate registry
- Creating images
- Managing images
- Managing image streams
- Using image streams with Kubernetes resources
- Triggering updates on image stream changes
- Image configuration resources
- Using templates
- Using Ruby on Rails
- Using images
Applications
- Applications overview
- Projects
- Application life cycle management
- Deployments
- Quotas
  - Resource quotas per project
  - Resource quotas across multiple projects
- Using config maps with applications
- Monitoring project and application metrics using the Developer perspective
- Monitoring application health
- Idling applications
- Pruning objects to reclaim resources
- Using the Red Hat Marketplace
Machine management
- Overview of machine management
- Creating machine sets
- Manually scaling a machine set
- Modifying a machine set
- Deleting a machine
- Applying autoscaling to a cluster
- Creating infrastructure machine sets
- User-provisioned infrastructure
- Deploying machine health checks
Nodes
- Overview of nodes
- Working with pods
- Controlling pod placement onto nodes (scheduling)
- Using Jobs and DaemonSets
  - Running background tasks on nodes automatically with daemonsets
  - Running tasks in pods using jobs
- Working with nodes
- Working with containers
- Working with clusters
Windows Container Support for OpenShift
- Red Hat OpenShift support for Windows Containers overview
- Red Hat OpenShift support for Windows Containers release notes
- Understanding Windows container workloads
- Enabling Windows container workloads
- Creating Windows MachineSet objects
  - Creating a Windows MachineSet object on AWS
  - Creating a Windows MachineSet object on Azure
- Scheduling Windows container workloads
- Windows node upgrades
- Removing Windows nodes
- Disabling Windows container workloads
Logging
- About cluster logging
- Installing cluster logging
- Configuring your cluster logging deployment
- Viewing logs for a specific resource
- Viewing cluster logs in Kibana
- Forwarding logs to third party systems
- Collecting and storing Kubernetes events
- Updating cluster logging
- Viewing cluster dashboards
- Troubleshooting cluster logging
- Uninstalling cluster logging
- Exported fields
Monitoring
- Monitoring overview
- Configuring the monitoring stack
- Enabling monitoring for user-defined projects
- Managing metrics
- Managing alerts
- Reviewing monitoring dashboards
- Accessing third-party UIs
- Exposing custom application metrics for autoscaling
- Troubleshooting monitoring issues
Metering
- About metering
- Installing metering
- Upgrading metering
- Configuring metering
- Reports
  - About reports
  - Storage Locations
- Using metering
- Examples of using metering
- Troubleshooting and debugging
- Uninstalling metering
Scalability and performance
- Recommended installation practices
- Recommended host practices
- Recommended cluster scaling practices
- Using the Node Tuning Operator
- Using Cluster Loader
- Using CPU Manager
- Using Topology Manager
- Scaling the Cluster Monitoring Operator
- Planning your environment according to object maximums
- Optimizing storage
- Optimizing routing
- Optimizing networking
- Managing bare metal hosts
- What huge pages do and how they are consumed by apps
- Optimizing data plane performance with Intel devices
Backup and restore
- Overview of backup and restore operations
- Shutting down a cluster gracefully
- Restarting a cluster gracefully
- Application backup and restore
- Control plane backup and restore
Migrating from version 3 to 4
- Migrating from version 3 to 4 overview
- About migrating from OKD 3 to 4
- Differences between OKD 3 and 4
- Planning considerations
- About MTC
- Installing MTC
- Installing MTC in a restricted network environment
- Upgrading MTC
- Premigration checklists
- Migrating your applications
- Advanced migration options
- Troubleshooting
Migration Toolkit for Containers
- Migration toolkit for containers overview
- About MTC
- MTC release notes
- Installing MTC
- Installing MTC in a restricted network environment
- Upgrading MTC
- Premigration checklists
- Migrating your applications
- Advanced migration options
- Troubleshooting
API reference
- Editing kubelet log level verbosity and gathering logs
- API list
- Common object reference
  - Index
- Authorization APIs
  - About Authorization APIs
  - LocalResourceAccessReview [authorization.openshift.io/v1]
  - LocalSubjectAccessReview [authorization.openshift.io/v1]
  - ResourceAccessReview [authorization.openshift.io/v1]
  - SelfSubjectRulesReview [authorization.openshift.io/v1]
  - SubjectAccessReview [authorization.openshift.io/v1]
  - SubjectRulesReview [authorization.openshift.io/v1]
  - TokenReview [authentication.k8s.io/v1]
  - LocalSubjectAccessReview [authorization.k8s.io/v1]
  - SelfSubjectAccessReview [authorization.k8s.io/v1]
  - SelfSubjectRulesReview [authorization.k8s.io/v1]
  - SubjectAccessReview [authorization.k8s.io/v1]
- Autoscale APIs
  - About Autoscale APIs
  - ClusterAutoscaler [autoscaling.openshift.io/v1]
  - MachineAutoscaler [autoscaling.openshift.io/v1beta1]
  - HorizontalPodAutoscaler [autoscaling/v1]
- Config APIs
  - About Config APIs
  - APIServer [config.openshift.io/v1]
  - Authentication [config.openshift.io/v1]
  - Build [config.openshift.io/v1]
  - ClusterOperator [config.openshift.io/v1]
  - ClusterVersion [config.openshift.io/v1]
  - Console [config.openshift.io/v1]
  - DNS [config.openshift.io/v1]
  - FeatureGate [config.openshift.io/v1]
  - HelmChartRepository [helm.openshift.io/v1beta1]
  - Image [config.openshift.io/v1]
  - Infrastructure [config.openshift.io/v1]
  - Ingress [config.openshift.io/v1]
  - Network [config.openshift.io/v1]
  - OAuth [config.openshift.io/v1]
  - OperatorHub [config.openshift.io/v1]
  - Project [config.openshift.io/v1]
  - Proxy [config.openshift.io/v1]
  - Scheduler [config.openshift.io/v1]
- Console APIs
  - About Console APIs
  - ConsoleCLIDownload [console.openshift.io/v1]
  - ConsoleExternalLogLink [console.openshift.io/v1]
  - ConsoleLink [console.openshift.io/v1]
  - ConsoleNotification [console.openshift.io/v1]
  - ConsoleYAMLSample [console.openshift.io/v1]
- Extension APIs
  - About Extension APIs
  - APIService [apiregistration.k8s.io/v1]
  - CustomResourceDefinition [apiextensions.k8s.io/v1]
  - MutatingWebhookConfiguration [admissionregistration.k8s.io/v1]
  - ValidatingWebhookConfiguration [admissionregistration.k8s.io/v1]
- Image APIs
  - About Image APIs
  - Image [image.openshift.io/v1]
  - ImageSignature [image.openshift.io/v1]
  - ImageStreamImage [image.openshift.io/v1]
  - ImageStreamImport [image.openshift.io/v1]
  - ImageStreamMapping [image.openshift.io/v1]
  - ImageStream [image.openshift.io/v1]
  - ImageStreamTag [image.openshift.io/v1]
  - ImageTag [image.openshift.io/v1]
- Machine APIs
  - About Machine APIs
  - ContainerRuntimeConfig [machineconfiguration.openshift.io/v1]
  - ControllerConfig [machineconfiguration.openshift.io/v1]
  - KubeletConfig [machineconfiguration.openshift.io/v1]
  - MachineConfigPool [machineconfiguration.openshift.io/v1]
  - MachineConfig [machineconfiguration.openshift.io/v1]
  - MachineHealthCheck [machine.openshift.io/v1beta1]
  - Machine [machine.openshift.io/v1beta1]
  - MachineSet [machine.openshift.io/v1beta1]
- Metadata APIs
  - About Metadata APIs
  - Binding [core/v1]
  - ComponentStatus [core/v1]
  - ConfigMap [core/v1]
  - ControllerRevision [apps/v1]
  - Event [events.k8s.io/v1]
  - Event [core/v1]
  - Lease [coordination.k8s.io/v1]
  - Namespace [core/v1]
- Monitoring APIs
  - About Monitoring APIs
  - Alertmanager [monitoring.coreos.com/v1]
  - PodMonitor [monitoring.coreos.com/v1]
  - Probe [monitoring.coreos.com/v1]
  - Prometheus [monitoring.coreos.com/v1]
  - PrometheusRule [monitoring.coreos.com/v1]
  - ServiceMonitor [monitoring.coreos.com/v1]
  - ThanosRuler [monitoring.coreos.com/v1]
- Network APIs
  - About Network APIs
  - ClusterNetwork [network.openshift.io/v1]
  - Endpoints [core/v1]
  - EndpointSlice [discovery.k8s.io/v1beta1]
  - EgressNetworkPolicy [network.openshift.io/v1]
  - HostSubnet [network.openshift.io/v1]
  - Ingress [networking.k8s.io/v1]
  - IngressClass [networking.k8s.io/v1]
  - IPPool [whereabouts.cni.cncf.io/v1alpha1]
  - NetNamespace [network.openshift.io/v1]
  - NetworkAttachmentDefinition [k8s.cni.cncf.io/v1]
  - NetworkPolicy [networking.k8s.io/v1]
  - Route [route.openshift.io/v1]
  - Service [core/v1]
- Node APIs
  - About Node APIs
  - Node [core/v1]
  - Profile [tuned.openshift.io/v1]
  - RuntimeClass [node.k8s.io/v1beta1]
  - Tuned [tuned.openshift.io/v1]
- OAuth APIs
  - About OAuth APIs
  - OAuthAccessToken [oauth.openshift.io/v1]
  - OAuthAuthorizeToken [oauth.openshift.io/v1]
  - OAuthClientAuthorization [oauth.openshift.io/v1]
  - OAuthClient [oauth.openshift.io/v1]
- Operator APIs
  - About Operator APIs
  - Authentication [operator.openshift.io/v1]
  - CloudCredential [operator.openshift.io/v1]
  - ClusterCSIDriver [operator.openshift.io/v1]
  - Console [operator.openshift.io/v1]
  - Config [operator.openshift.io/v1]
  - Config [imageregistry.operator.openshift.io/v1]
  - Config [samples.operator.openshift.io/v1]
  - CSISnapshotController [operator.openshift.io/v1]
  - DNS [operator.openshift.io/v1]
  - DNSRecord [ingress.operator.openshift.io/v1]
  - Etcd [operator.openshift.io/v1]
  - ImageContentSourcePolicy [operator.openshift.io/v1alpha1]
  - ImagePruner [imageregistry.operator.openshift.io/v1]
  - IngressController [operator.openshift.io/v1]
  - KubeAPIServer [operator.openshift.io/v1]
  - KubeControllerManager [operator.openshift.io/v1]
  - KubeScheduler [operator.openshift.io/v1]
  - KubeStorageVersionMigrator [operator.openshift.io/v1]
  - Network [operator.openshift.io/v1]
  - OpenShiftAPIServer [operator.openshift.io/v1]
  - OpenShiftControllerManager [operator.openshift.io/v1]
  - OperatorPKI [network.operator.openshift.io/v1]
  - ServiceCA [operator.openshift.io/v1]
  - Storage [operator.openshift.io/v1]
- OperatorHub APIs
  - About OperatorHub APIs
  - CatalogSource [operators.coreos.com/v1alpha1]
  - ClusterServiceVersion [operators.coreos.com/v1alpha1]
  - InstallPlan [operators.coreos.com/v1alpha1]
  - OperatorGroup [operators.coreos.com/v1]
  - PackageManifest [packages.operators.coreos.com/v1]
  - Subscription [operators.coreos.com/v1alpha1]
- Policy APIs
  - About Policy APIs
  - PodDisruptionBudget [policy/v1beta1]
- Project APIs
  - About Project APIs
  - Project [project.openshift.io/v1]
  - ProjectRequest [project.openshift.io/v1]
- Provisioning APIs
  - About Provisioning APIs
  - BareMetalHost [metal3.io/v1alpha1]
  - Provisioning [metal3.io/v1alpha1]
- RBAC APIs
  - About RBAC APIs
  - ClusterRoleBinding [rbac.authorization.k8s.io/v1]
  - ClusterRole [rbac.authorization.k8s.io/v1]
  - RoleBinding [rbac.authorization.k8s.io/v1]
  - Role [rbac.authorization.k8s.io/v1]
- Role APIs
  - About Role APIs
  - ClusterRoleBinding [authorization.openshift.io/v1]
  - ClusterRole [authorization.openshift.io/v1]
  - RoleBindingRestriction [authorization.openshift.io/v1]
  - RoleBinding [authorization.openshift.io/v1]
  - Role [authorization.openshift.io/v1]
- Schedule and quota APIs
  - About Schedule and quota APIs
  - AppliedClusterResourceQuota [quota.openshift.io/v1]
  - ClusterResourceQuota [quota.openshift.io/v1]
  - FlowSchema [flowcontrol.apiserver.k8s.io/v1alpha1]
  - LimitRange [core/v1]
  - PriorityClass [scheduling.k8s.io/v1]
  - PriorityLevelConfiguration [flowcontrol.apiserver.k8s.io/v1alpha1]
  - ResourceQuota [core/v1]
- Security APIs
  - About Security APIs
  - CertificateSigningRequest [certificates.k8s.io/v1]
  - CredentialsRequest [cloudcredential.openshift.io/v1]
  - PodSecurityPolicyReview [security.openshift.io/v1]
  - PodSecurityPolicySelfSubjectReview [security.openshift.io/v1]
  - PodSecurityPolicySubjectReview [security.openshift.io/v1]
  - RangeAllocation [security.openshift.io/v1]
  - Secret [core/v1]
  - SecurityContextConstraints [security.openshift.io/v1]
  - ServiceAccount [core/v1]
- Storage APIs
  - About Storage APIs
  - CSIDriver [storage.k8s.io/v1]
  - CSINode [storage.k8s.io/v1]
  - PersistentVolumeClaim [core/v1]
  - StorageClass [storage.k8s.io/v1]
  - StorageState [migration.k8s.io/v1alpha1]
  - StorageVersionMigration [migration.k8s.io/v1alpha1]
  - VolumeAttachment [storage.k8s.io/v1]
  - VolumeSnapshot [snapshot.storage.k8s.io/v1beta1]
  - VolumeSnapshotClass [snapshot.storage.k8s.io/v1beta1]
  - VolumeSnapshotContent [snapshot.storage.k8s.io/v1beta1]
- Template APIs
  - About Template APIs
  - BrokerTemplateInstance [template.openshift.io/v1]
  - PodTemplate [core/v1]
  - Template [template.openshift.io/v1]
  - TemplateInstance [template.openshift.io/v1]
- User and group APIs
  - About User and group APIs
  - Group [user.openshift.io/v1]
  - Identity [user.openshift.io/v1]
  - UserIdentityMapping [user.openshift.io/v1]
  - User [user.openshift.io/v1]
- Workloads APIs
  - About Workloads APIs
  - BuildConfig [build.openshift.io/v1]
  - Build [build.openshift.io/v1]
  - CronJob [batch/v1beta1]
  - DaemonSet [apps/v1]
  - Deployment [apps/v1]
  - DeploymentConfig [apps.openshift.io/v1]
  - Job [batch/v1]
  - Pod [core/v1]
  - ReplicationController [core/v1]
  - PersistentVolume [core/v1]
  - ReplicaSet [apps/v1]
  - StatefulSet [apps/v1]

Troubleshooting the installer workflow
Troubleshooting install-config.yaml
Bootstrap VM issues
- Bootstrap VM cannot boot up the cluster nodes
- Inspecting logs
Cluster nodes will not PXE boot
The API is not accessible
Cleaning up previous installations
Issues with creating the registry
Miscellaneous issues
Reviewing the installation

Troubleshooting the installer workflow

Prior to troubleshooting the installation environment, it is critical to understand the overall flow of the installer-provisioned installation on bare metal. The diagrams below provide a troubleshooting flow with a step-by-step breakdown for the environment.

Flow-Diagram-1

Workflow 1 of 4 illustrates a troubleshooting workflow when the install-config.yaml file has errors or the Fedora CoreOS (FCOS) images are inaccessible. Troubleshooting suggestions can be found at Troubleshooting install-config.yaml.

Flow-Diagram-2

Workflow 2 of 4 illustrates a troubleshooting workflow for bootstrap VM issues, bootstrap VMs that cannot boot up the cluster nodes, and inspecting logs. When installing a OKD cluster without the provisioning network, this workflow does not apply.