Anyone good with Bash heredocs?
-
I'm trying to use a neat bash script but the project is semi-abandoned and it doesn't work on my system (CentOS 6.5). Aparently it does work for some people so it seems to be some sort of portability issue. I've fixed some other bugs within my skill range but I'm only barely aware of how heredocs work so this is stumping me.
[code]
line 522: syntax error near unexpected token `<'
[/code]The line in question:
[code]
while read -r; do tmp[z++]="${db}.${REPLY}"; done < <(mysql --user="${CONFIG_mysql_dump_username}" --password="${CONFIG_mysql_dump_password}" --host="${CONFIG_mysql_dump_host}" "${mysql_opt[@]}" --batch --skip-column-names -e "select table_name from information_schema.tables where table_schema='${db}' and table_name like '${table//*/%}';")
[/code]That seems to me like a heredoc using parenthesis as the delimiter? I tried fixing the whitespace between the '<' and changing the delimiters in case those were the the issues but no dice.
Any ideas?
-
Does this work?
mysql --user="${CONFIG_mysql_dump_username}" --password="${CONFIG_mysql_dump_password}" --host="${CONFIG_mysql_dump_host}" "${mysql_opt[@]}" --batch --skip-column-names -e "select table_name from information_schema.tables where table_schema='${db}' and table_name like '${table//\*/%}';"
What's the output of this?
bash --version
-
Thanks for the help. This is the output of bash --version:
[code]
GNU bash, version 4.1.2(1)-release (x86_64-redhat-linux-gnu)
Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.htmlThis is free software; you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
[/code]I'm just working on dumping those variables so I can use exactly what it's trying to do, I'll edit this when I have output in a sec...
Edit: See below, can't get output. Unsure why...
-
Strange, that's a good bash version. I don't immediately see how that read command works, but it shouldn't produce that error.
Maybe you have invisible characters?
Or some alias? Try executing script in a clean environment:
env -i bash --noprofile --norc
-
Or some alias? Try executing script in a clean environment:
Exactly the same result unfortunately. Something weird is going on, I can't get echo to work for peeking at the variables. I disabled the script's redirection without joy and then tried the simplest thing I could think of: [code]echo "test" &>/dev/tty[/code]
Fairly sure that should always work?
Edit: [echo "test" &>/dev/tty] works fine at the shell prompt, it does not work from within the script. Just to make that clear!
-
Fairly sure that should always work?
Yup, tried on Centos 6, same bash version as yours.
Something is fucked up with your machine. No clue what.
I'm afraid you'll need to upgrade to a higher tier of tech support. :-)
-
I'm afraid you'll need to upgrade to a higher tier of tech support
Thanks a lot for the help anyway, much appreciated!
-
Are you running the script via
#!/bin/sh
,#!/bin/bash
, or something else?EDIT: also, that is process substitution, not heredoc.
<(foo)
is replaced by a named pipe, reading from that pipe will return the output of the commandfoo
.
-
also, that is process substitution, not heredoc.
This.
Heredocs are indicated by the single token
<<
, you have a space between your two making them two separate tokens (<
redirection and<(
which forms part of a<( command )
temporary pipe)..[Heredocs][1]:
#!/bin/bash cat <<RanDomString -------------------------- The quick brown fox jumped over the lazy dog -------------------------- RanDomString
[Temporary pipes][2]:
# diff <(cat file_a) <(cat file_b)
```
while read -r; do tmp[z++]="${db}.${REPLY}"; done < <(mysql --user="${CONFIG_...Anyway, not sure what's wrong with yours - a changed version to test it on one of my systems seems to work fine:
-bash-4.1$ while read -r; do echo "psa".${REPLY}; done < <(mysql -u admin -p
cat ~/private/.mysqlpass
--batch --skip-column-names -e 'select table_name from information_schema.tables where table_schema like "psa"')
psa.APSApplicationItems
psa.APSCatalogUpdates
psa.APSClientApplicationItems
psa.APSLicenseTypes
psa.APSLicenses
psa.ApiRpcCallsStat
psa.BackendCache
psa.BackupsScheduled
psa.BackupsSettings
psa.Cards
psa.ClientsTraffic
psa.Components
psa.Configurations
psa.DashboardPreset
psa.DashboardPresetConfig
psa.DatabaseServers
psa.DomainServices
psa.DomainsTraffic
psa.GL_remote_domains
psa.GL_settings
psa.IP_Addresses
psa.IpAddressesCollections
psa.IpCollections
psa.Limits
psa.Logos
psa.MailLists
psa.MailMessagesStat
psa.ModuleSettings
psa.Modules
psa.Notes
psa.Notifications
psa.PMM
psa.PMMDefault
psa.Parameters
psa.Permissions
psa.PersistentCache
psa.PhpSettings
psa.PhpSettingsParameters
psa.PlanItemProperties
psa.PlanItems
psa.PlansSubscriptions
psa.PleskPagesStat
psa.Repository
psa.SBConfig
psa.SBResellers
psa.SBSites
psa.SSOBranding
psa.ServiceNodeCache
psa.ServiceNodeCertificates
psa.ServiceNodeProperties
psa.ServiceNodes
psa.SiteAppFiles
psa.SiteAppPackages
psa.SiteAppResources
psa.SiteApps
psa.SiteAppsHitsStat
psa.SitePagesStat
psa.Skins
psa.SubscriptionProperties
psa.Subscriptions
psa.Templates
psa.TmplData
psa.WebApps
psa.Webmails
psa.accounts
psa.actions
psa.ai_vendor_sources
psa.anon_ftp
psa.apsContexts
psa.apsContextsApplications
psa.apsResources
psa.apsResourcesParameters
psa.apscategories
psa.badmailfrom
psa.certificates
psa.cl_param
psa.clients
psa.cp_access
psa.custom_buttons
psa.data_bases
psa.db_users
psa.disk_usage
psa.dns_recs
psa.dns_recs_t
psa.dns_refs
psa.dns_zone
psa.dom_level_usrs
psa.dom_param
psa.domainaliases
psa.domains
psa.event_handlers
psa.exp_event
psa.externalWebmails
psa.fileSharingUsers
psa.forwarding
psa.ftp_users
psa.hosting
psa.ip_pool
psa.itmpl
psa.itmpl_data
psa.key_history
psa.key_history_params
psa.locales
psa.log_actions
psa.log_components
psa.log_rotation
psa.longtaskparams
psa.longtasks
psa.mail
psa.mail_aliases
psa.mail_redir
psa.mail_resp
psa.mass_mail
psa.mass_mail_clients
psa.mass_mail_domains
psa.migration_version
psa.misc
psa.mn_param
psa.password_secrets
psa.pd_users
psa.protected_dirs
psa.report
psa.report_auto
psa.report_section
psa.resp_attach
psa.resp_forward
psa.resp_freq
psa.secret_keys
psa.sessions
psa.siteapppackages_apscategories
psa.smb_apsBundleFilterItems
psa.smb_apsBundleFilters
psa.smb_apsCategories
psa.smb_apsContexts
psa.smb_apsImportedResources
psa.smb_apsImportedSettings
psa.smb_apsInstanceErrors
psa.smb_apsInstances
psa.smb_apsMetas
psa.smb_apsPackageUpdates
psa.smb_apsPackages
psa.smb_apsPackagesCategories
psa.smb_apsProvisionEnvironments
psa.smb_apsProvisions
psa.smb_apsResourceParameters
psa.smb_apsResources
psa.smb_apsSettings
psa.smb_componentUpdates
psa.smb_emailAliases
psa.smb_fileSharingUnlistedFiles
psa.smb_generalPermissions
psa.smb_locales
psa.smb_productUpgrades
psa.smb_roleGeneralPermissions
psa.smb_roleServicePermissions
psa.smb_roles
psa.smb_serviceEntryPoints
psa.smb_serviceInstances
psa.smb_servicePermissionAccounts
psa.smb_servicePermissions
psa.smb_serviceProviders
psa.smb_settings
psa.smb_userServicePermissions
psa.smb_users
psa.smtp_poplocks
psa.spamfilter
psa.spamfilter_preferences
psa.subdomains
psa.suspend_handler_history
psa.sys_users
psa.upgrade_history
psa.web_users
psa.webalizer_group_referrer
psa.webalizer_hidden_referrer
-bash-4.1$-bash-4.1$ bash --version
GNU bash, version 4.1.2(1)-release (x86_64-redhat-linux-gnu)
Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.htmlThis is free software; you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
-bash-4.1$-bash-4.1$ uname -a
Linux s15474768.onlinehome-server.info 2.6.32-042stab108.2 #1 SMP Tue May 12 18:07:50 MSK 2015 x86_64 x86_64 x86_64 GNU/Linux[1]: http://ss64.com/bash/syntax-here.html [2]: https://crashingdaily.wordpress.com/2008/03/06/diff-two-stdout-streams/
-
Another system:
[pjh@thinkpad ~]$ while read -r; do echo "information_schema".${REPLY}; done < <(mysql -u root -p`cat ~/private/.mysqlpass` --batch --skip-column-names -e 'select table_name from information_schema.tables where table_schema like "information_schema"') information_schema.CHARACTER_SETS information_schema.COLLATIONS information_schema.COLLATION_CHARACTER_SET_APPLICABILITY information_schema.COLUMNS information_schema.COLUMN_PRIVILEGES information_schema.ENGINES information_schema.EVENTS information_schema.FILES information_schema.GLOBAL_STATUS information_schema.GLOBAL_VARIABLES information_schema.KEY_COLUMN_USAGE information_schema.PARTITIONS information_schema.PLUGINS information_schema.PROCESSLIST information_schema.PROFILING information_schema.REFERENTIAL_CONSTRAINTS information_schema.ROUTINES information_schema.SCHEMATA information_schema.SCHEMA_PRIVILEGES information_schema.SESSION_STATUS information_schema.SESSION_VARIABLES information_schema.STATISTICS information_schema.TABLES information_schema.TABLE_CONSTRAINTS information_schema.TABLE_PRIVILEGES information_schema.TRIGGERS information_schema.USER_PRIVILEGES information_schema.VIEWS
[pjh@thinkpad ~]$ bash --version GNU bash, version 4.3.33(1)-release (x86_64-mandriva-linux-gnu) Copyright (C) 2013 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software; you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law.
[pjh@thinkpad ~]$ uname -a Linux thinkpad.pjh 3.2.18-pclos2.bfs #1 SMP PREEMPT Thu May 24 12:11:06 CEST 2012 x86_64 x86_64 x86_64 GNU/Linux
-
that is process substitution, not heredoc
@PJH said:This.
Thank you both very much for the info. I wasn't aware that was a thing, I'll have to do some reading.
Are you running the script via #!/bin/sh, #!/bin/bash, or something else?
I was running it via sh which it turns out is NOT symlinked to bash on CentOS...
Running via bash gives me no errors! No output either but no errors, I can work with that
-
Note on my RHEL5, /bin/sh is symlinked to /bin/bash, but bash does not support the
<()
construct when invoked as sh.
-
I wasn't aware that was a thing, I'll have to do some reading.
The usual use of process substitution is to make a named pipe with a process attached to the other end available as a command line argument, as in the diff example above. Using it as a redirection target the way that script does is kind of weird; it just looks like a really convoluted way to avoid using a pipe, like this:
mysql \ --user="${CONFIG_mysql_dump_username}" \ --password="${CONFIG_mysql_dump_password}" \ --host="${CONFIG_mysql_dump_host}" \ "${mysql_opt[@]}" \ --batch \ --skip-column-names \ -e "select table_name from information_schema.tables where table_schema='${db}' and table_name like '${table//\*/%}';" \ | while read -r do tmp[z++]="${db}.${REPLY}" done
The pipeline version still won't run in shells that don't support array variables, and also relies on REPLY as the default target of a read command, which is a bashism.
Edit: just looking at the pipeline version all laid out, I see why they've done the process substitution thing. Pipelined commands each run in a subshell, which means that the
tmp
array accumulated inside thewhile read
loop would evaporate as soon as the loop ends. The way they've done it, thewhile read
loop runs in the outer shell and only themysql
command goes in a subshell.If you want to do that in a shell that doesn't support process substitution, you can actually get it done with a here document, using ordinary command substitution:
while read -r do tmp[z++]="${db}.${REPLY}" done <<EOF $(mysql \ --user="${CONFIG_mysql_dump_username}" \ --password="${CONFIG_mysql_dump_password}" \ --host="${CONFIG_mysql_dump_host}" \ "${mysql_opt[@]}" \ --batch \ --skip-column-names \ -e "select table_name from information_schema.tables where table_schema='${db}' and table_name like '${table//\*/%}';" \ ) EOF
Here documents are actually built up as temporary files, so this version would run the
mysql
query to completion before making any of its output available to thewhile read
loop. The named pipe created by the process substitution version would let both run in parallel.
-
process substitution
process substitution
process substitution
Could everybody please start referring to this as ‘the Kirby operator’, because a) that's what it does, right—it sucks everything up and then transforms into it—and b)
<('.'<)
.
-
No. But you're absolutely right.
-
the Kirby operator...
<('.'<)
I don't get it.
-
The named pipe created by the process substitution version would let both run in parallel.
That's actually very cool, I'll have to stop thinking of scripting as inherently a sequential system.
Thanks for the detailed explanation as well, it makes sense why you would want to use the different methods now.
-
I'll have to stop thinking of scripting as inherently a sequential system.
For what it's worth, all the processes in a normal pipeline run in parallel as well; this is true for Unix scripts and for Windows cmd and Powershell scripts too.
-
... but there's some things that you can depend on happening predictably (maybe not when you like, but predictably) - like when the standard streams are opened.
Sorry it was just too scary without that.