Anyone good with Bash heredocs?



  • I'm trying to use a neat bash script but the project is semi-abandoned and it doesn't work on my system (CentOS 6.5). Aparently it does work for some people so it seems to be some sort of portability issue. I've fixed some other bugs within my skill range but I'm only barely aware of how heredocs work so this is stumping me.

    [code]
    line 522: syntax error near unexpected token `<'
    [/code]

    The line in question:

    [code]
    while read -r; do tmp[z++]="${db}.${REPLY}"; done < <(mysql --user="${CONFIG_mysql_dump_username}" --password="${CONFIG_mysql_dump_password}" --host="${CONFIG_mysql_dump_host}" "${mysql_opt[@]}" --batch --skip-column-names -e "select table_name from information_schema.tables where table_schema='${db}' and table_name like '${table//*/%}';")
    [/code]

    That seems to me like a heredoc using parenthesis as the delimiter? I tried fixing the whitespace between the '<' and changing the delimiters in case those were the the issues but no dice.

    Any ideas?



  • Does this work?

    mysql --user="${CONFIG_mysql_dump_username}" --password="${CONFIG_mysql_dump_password}" --host="${CONFIG_mysql_dump_host}" "${mysql_opt[@]}" --batch --skip-column-names -e "select table_name from information_schema.tables where table_schema='${db}' and table_name like '${table//\*/%}';"
    

    What's the output of this?

    bash --version
    


  • Thanks for the help. This is the output of bash --version:

    [code]
    GNU bash, version 4.1.2(1)-release (x86_64-redhat-linux-gnu)
    Copyright (C) 2009 Free Software Foundation, Inc.
    License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html

    This is free software; you are free to change and redistribute it.
    There is NO WARRANTY, to the extent permitted by law.
    [/code]

    I'm just working on dumping those variables so I can use exactly what it's trying to do, I'll edit this when I have output in a sec...

    Edit: See below, can't get output. Unsure why...



  • Strange, that's a good bash version. I don't immediately see how that read command works, but it shouldn't produce that error.

    Maybe you have invisible characters?

    Or some alias? Try executing script in a clean environment:

    env -i bash --noprofile --norc
    


  • @cartman82 said:

    Or some alias? Try executing script in a clean environment:

    Exactly the same result unfortunately. Something weird is going on, I can't get echo to work for peeking at the variables. I disabled the script's redirection without joy and then tried the simplest thing I could think of: [code]echo "test" &>/dev/tty[/code]

    Fairly sure that should always work?

    Edit: [echo "test" &>/dev/tty] works fine at the shell prompt, it does not work from within the script. Just to make that clear!



  • @Cursorkeys said:

    Fairly sure that should always work?

    Yup, tried on Centos 6, same bash version as yours.

    Something is fucked up with your machine. No clue what.

    I'm afraid you'll need to upgrade to a higher tier of tech support. :-)



  • @cartman82 said:

    I'm afraid you'll need to upgrade to a higher tier of tech support

    Thanks a lot for the help anyway, much appreciated!



  • Are you running the script via #!/bin/sh, #!/bin/bash, or something else?

    EDIT: also, that is process substitution, not heredoc. <(foo) is replaced by a named pipe, reading from that pipe will return the output of the command foo.



  • @PleegWat said:

    also, that is process substitution, not heredoc.

    This.

    Heredocs are indicated by the single token <<, you have a space between your two making them two separate tokens (< redirection and <( which forms part of a <( command ) temporary pipe)..

    [Heredocs][1]:

    #!/bin/bash
    cat <<RanDomString
    --------------------------
    The quick brown fox 
    jumped over the lazy dog 
    --------------------------
    RanDomString
    

    [Temporary pipes][2]:

    # diff <(cat file_a) <(cat file_b)
    

    @Cursorkeys said:

    ```
    while read -r; do tmp[z++]="${db}.${REPLY}"; done < <(mysql --user="${CONFIG_...

    
    Anyway, not sure what's wrong with yours - a changed version to test it on one of my systems seems to work fine:
    
    

    -bash-4.1$ while read -r; do echo "psa".${REPLY}; done < <(mysql -u admin -pcat ~/private/.mysqlpass --batch --skip-column-names -e 'select table_name from information_schema.tables where table_schema like "psa"')
    psa.APSApplicationItems
    psa.APSCatalogUpdates
    psa.APSClientApplicationItems
    psa.APSLicenseTypes
    psa.APSLicenses
    psa.ApiRpcCallsStat
    psa.BackendCache
    psa.BackupsScheduled
    psa.BackupsSettings
    psa.Cards
    psa.ClientsTraffic
    psa.Components
    psa.Configurations
    psa.DashboardPreset
    psa.DashboardPresetConfig
    psa.DatabaseServers
    psa.DomainServices
    psa.DomainsTraffic
    psa.GL_remote_domains
    psa.GL_settings
    psa.IP_Addresses
    psa.IpAddressesCollections
    psa.IpCollections
    psa.Limits
    psa.Logos
    psa.MailLists
    psa.MailMessagesStat
    psa.ModuleSettings
    psa.Modules
    psa.Notes
    psa.Notifications
    psa.PMM
    psa.PMMDefault
    psa.Parameters
    psa.Permissions
    psa.PersistentCache
    psa.PhpSettings
    psa.PhpSettingsParameters
    psa.PlanItemProperties
    psa.PlanItems
    psa.PlansSubscriptions
    psa.PleskPagesStat
    psa.Repository
    psa.SBConfig
    psa.SBResellers
    psa.SBSites
    psa.SSOBranding
    psa.ServiceNodeCache
    psa.ServiceNodeCertificates
    psa.ServiceNodeProperties
    psa.ServiceNodes
    psa.SiteAppFiles
    psa.SiteAppPackages
    psa.SiteAppResources
    psa.SiteApps
    psa.SiteAppsHitsStat
    psa.SitePagesStat
    psa.Skins
    psa.SubscriptionProperties
    psa.Subscriptions
    psa.Templates
    psa.TmplData
    psa.WebApps
    psa.Webmails
    psa.accounts
    psa.actions
    psa.ai_vendor_sources
    psa.anon_ftp
    psa.apsContexts
    psa.apsContextsApplications
    psa.apsResources
    psa.apsResourcesParameters
    psa.apscategories
    psa.badmailfrom
    psa.certificates
    psa.cl_param
    psa.clients
    psa.cp_access
    psa.custom_buttons
    psa.data_bases
    psa.db_users
    psa.disk_usage
    psa.dns_recs
    psa.dns_recs_t
    psa.dns_refs
    psa.dns_zone
    psa.dom_level_usrs
    psa.dom_param
    psa.domainaliases
    psa.domains
    psa.event_handlers
    psa.exp_event
    psa.externalWebmails
    psa.fileSharingUsers
    psa.forwarding
    psa.ftp_users
    psa.hosting
    psa.ip_pool
    psa.itmpl
    psa.itmpl_data
    psa.key_history
    psa.key_history_params
    psa.locales
    psa.log_actions
    psa.log_components
    psa.log_rotation
    psa.longtaskparams
    psa.longtasks
    psa.mail
    psa.mail_aliases
    psa.mail_redir
    psa.mail_resp
    psa.mass_mail
    psa.mass_mail_clients
    psa.mass_mail_domains
    psa.migration_version
    psa.misc
    psa.mn_param
    psa.password_secrets
    psa.pd_users
    psa.protected_dirs
    psa.report
    psa.report_auto
    psa.report_section
    psa.resp_attach
    psa.resp_forward
    psa.resp_freq
    psa.secret_keys
    psa.sessions
    psa.siteapppackages_apscategories
    psa.smb_apsBundleFilterItems
    psa.smb_apsBundleFilters
    psa.smb_apsCategories
    psa.smb_apsContexts
    psa.smb_apsImportedResources
    psa.smb_apsImportedSettings
    psa.smb_apsInstanceErrors
    psa.smb_apsInstances
    psa.smb_apsMetas
    psa.smb_apsPackageUpdates
    psa.smb_apsPackages
    psa.smb_apsPackagesCategories
    psa.smb_apsProvisionEnvironments
    psa.smb_apsProvisions
    psa.smb_apsResourceParameters
    psa.smb_apsResources
    psa.smb_apsSettings
    psa.smb_componentUpdates
    psa.smb_emailAliases
    psa.smb_fileSharingUnlistedFiles
    psa.smb_generalPermissions
    psa.smb_locales
    psa.smb_productUpgrades
    psa.smb_roleGeneralPermissions
    psa.smb_roleServicePermissions
    psa.smb_roles
    psa.smb_serviceEntryPoints
    psa.smb_serviceInstances
    psa.smb_servicePermissionAccounts
    psa.smb_servicePermissions
    psa.smb_serviceProviders
    psa.smb_settings
    psa.smb_userServicePermissions
    psa.smb_users
    psa.smtp_poplocks
    psa.spamfilter
    psa.spamfilter_preferences
    psa.subdomains
    psa.suspend_handler_history
    psa.sys_users
    psa.upgrade_history
    psa.web_users
    psa.webalizer_group_referrer
    psa.webalizer_hidden_referrer
    -bash-4.1$

    
    

    -bash-4.1$ bash --version
    GNU bash, version 4.1.2(1)-release (x86_64-redhat-linux-gnu)
    Copyright (C) 2009 Free Software Foundation, Inc.
    License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html

    This is free software; you are free to change and redistribute it.
    There is NO WARRANTY, to the extent permitted by law.
    -bash-4.1$

    -bash-4.1$ uname -a
    Linux s15474768.onlinehome-server.info 2.6.32-042stab108.2 #1 SMP Tue May 12 18:07:50 MSK 2015 x86_64 x86_64 x86_64 GNU/Linux

    
    
      [1]: http://ss64.com/bash/syntax-here.html
      [2]: https://crashingdaily.wordpress.com/2008/03/06/diff-two-stdout-streams/


  • Another system:

    [pjh@thinkpad ~]$ while read -r; do echo "information_schema".${REPLY}; done < <(mysql -u root -p`cat ~/private/.mysqlpass` --batch --skip-column-names -e 'select table_name from information_schema.tables where table_schema like "information_schema"')
    information_schema.CHARACTER_SETS
    information_schema.COLLATIONS
    information_schema.COLLATION_CHARACTER_SET_APPLICABILITY
    information_schema.COLUMNS
    information_schema.COLUMN_PRIVILEGES
    information_schema.ENGINES
    information_schema.EVENTS
    information_schema.FILES
    information_schema.GLOBAL_STATUS
    information_schema.GLOBAL_VARIABLES
    information_schema.KEY_COLUMN_USAGE
    information_schema.PARTITIONS
    information_schema.PLUGINS
    information_schema.PROCESSLIST
    information_schema.PROFILING
    information_schema.REFERENTIAL_CONSTRAINTS
    information_schema.ROUTINES
    information_schema.SCHEMATA
    information_schema.SCHEMA_PRIVILEGES
    information_schema.SESSION_STATUS
    information_schema.SESSION_VARIABLES
    information_schema.STATISTICS
    information_schema.TABLES
    information_schema.TABLE_CONSTRAINTS
    information_schema.TABLE_PRIVILEGES
    information_schema.TRIGGERS
    information_schema.USER_PRIVILEGES
    information_schema.VIEWS
    
    [pjh@thinkpad ~]$ bash --version
    GNU bash, version 4.3.33(1)-release (x86_64-mandriva-linux-gnu)
    Copyright (C) 2013 Free Software Foundation, Inc.
    License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
    
    This is free software; you are free to change and redistribute it.
    There is NO WARRANTY, to the extent permitted by law.
    
    [pjh@thinkpad ~]$ uname -a
    Linux thinkpad.pjh 3.2.18-pclos2.bfs #1 SMP PREEMPT Thu May 24 12:11:06 CEST 2012 x86_64 x86_64 x86_64 GNU/Linux
    


  • @PleegWat said:

    that is process substitution, not heredoc

    @PJH said:
    This.

    Thank you both very much for the info. I wasn't aware that was a thing, I'll have to do some reading.

    @PleegWat said:

    Are you running the script via #!/bin/sh, #!/bin/bash, or something else?

    :headdesk: I was running it via sh which it turns out is NOT symlinked to bash on CentOS...

    Running via bash gives me no errors! No output either but no errors, I can work with that :smile:



  • Note on my RHEL5, /bin/sh is symlinked to /bin/bash, but bash does not support the <() construct when invoked as sh.



  • @Cursorkeys said:

    I wasn't aware that was a thing, I'll have to do some reading.

    The usual use of process substitution is to make a named pipe with a process attached to the other end available as a command line argument, as in the diff example above. Using it as a redirection target the way that script does is kind of weird; it just looks like a really convoluted way to avoid using a pipe, like this:

    mysql \
        --user="${CONFIG_mysql_dump_username}" \
        --password="${CONFIG_mysql_dump_password}" \
        --host="${CONFIG_mysql_dump_host}" \
        "${mysql_opt[@]}" \
        --batch \
        --skip-column-names \
        -e "select table_name from information_schema.tables where table_schema='${db}' and table_name like '${table//\*/%}';" \
    |
    while read -r
    do
        tmp[z++]="${db}.${REPLY}"
    done
    

    The pipeline version still won't run in shells that don't support array variables, and also relies on REPLY as the default target of a read command, which is a bashism.

    Edit: just looking at the pipeline version all laid out, I see why they've done the process substitution thing. Pipelined commands each run in a subshell, which means that the tmp array accumulated inside the while read loop would evaporate as soon as the loop ends. The way they've done it, the while read loop runs in the outer shell and only the mysql command goes in a subshell.

    If you want to do that in a shell that doesn't support process substitution, you can actually get it done with a here document, using ordinary command substitution:

    while read -r
    do
        tmp[z++]="${db}.${REPLY}"
    done <<EOF
    $(mysql \
        --user="${CONFIG_mysql_dump_username}" \
        --password="${CONFIG_mysql_dump_password}" \
        --host="${CONFIG_mysql_dump_host}" \
        "${mysql_opt[@]}" \
        --batch \
        --skip-column-names \
        -e "select table_name from information_schema.tables where table_schema='${db}' and table_name like '${table//\*/%}';" \
    )
    EOF
    

    Here documents are actually built up as temporary files, so this version would run the mysql query to completion before making any of its output available to the while read loop. The named pipe created by the process substitution version would let both run in parallel.



  • @flabdablet said:

    process substitution

    @PJH said:

    process substitution

    @PleegWat said:

    process substitution

    Could everybody please start referring to this as ‘the Kirby operator’, because a) that's what it does, right—it sucks everything up and then transforms into it—and b) <('.'<).



  • No. But you're absolutely right.



  • @Buddy said:

    the Kirby operator...<('.'<)

    I don't get it.



  • @flabdablet said:

    The named pipe created by the process substitution version would let both run in parallel.

    That's actually very cool, I'll have to stop thinking of scripting as inherently a sequential system.

    Thanks for the detailed explanation as well, it makes sense why you would want to use the different methods now.



  • @Cursorkeys said:

    I'll have to stop thinking of scripting as inherently a sequential system.

    For what it's worth, all the processes in a normal pipeline run in parallel as well; this is true for Unix scripts and for Windows cmd and Powershell scripts too.



  • ... but there's some things that you can depend on happening predictably (maybe not when you like, but predictably) - like when the standard streams are opened.

    Sorry it was just too scary without that.


Log in to reply
 

Looks like your connection to What the Daily WTF? was lost, please wait while we try to reconnect.